Skip to content

feat(org-spec-audit): one question, one issue — the ledger becomes the artifact - #142

Merged
debuggingfuture merged 2 commits into
mainfrom
org-spec-audit/questions-as-issues
Aug 17, 2026
Merged

feat(org-spec-audit): one question, one issue — the ledger becomes the artifact#142
debuggingfuture merged 2 commits into
mainfrom
org-spec-audit/questions-as-issues

Conversation

@debuggingfuture

Copy link
Copy Markdown
Member

org-spec-audit files one GitHub issue per open question in the control repo instead of a dated markdown file in a daily draft PR. A sweep that re-finds a question already on file writes nothing — no comment, no reopen, no second issue. Implements the design in fractalboxdev/org#150.

Problem & Insight

The sweep produced org#147 and org#148 two days apart, four questions each, three of them the same question twice — two under a byte-identical maintenance-key.

An identical key that re-proposes rules out the obvious cause and leaves the real one. The ledger could only remember "no". It had no way to remember "asked". Both checkSuppression rules key off a terminated proposal — a decline in declined.jsonl, or a PR closed unmerged — and the first PR was neither. It was open, unanswered, in the queue, which the design read as no signal at all, so the second sweep re-proposed a question whose first proposal was still on screen, exactly as specified.

The unstable key is the second cause, not the first: the same question minted authorize-pipeline-dags one day and adopt-pipeline-dags the next. A verb encodes the proposed action, which is phrasing; the subject is not. Fixing only that would have fixed one duplicate of three.

Take

An issue already is what the suppression machinery was approximating, and it holds the state a ledger of declines structurally cannot: open answers "have I raised this?". One field, three meanings, no second file. open → write nothing; closed → never re-file, never reopen; absent → file.

Decisions worth review:

  • openIssue fails rather than degrading to a logged no-op, unlike openDraftPullRequest. A PR write that no-ops loses nothing — idempotent branch, content in git — while a no-op here drops the only copy of the question.
  • The dedup read fails closed, and runs before the sweep. It inverts the suppression primitive on purpose: an unreadable ledger cost one duplicate PR; an unreadable issue set costs a duplicate of every question at once. Placed ahead of the sweep so its failure doesn't waste an estate of model calls. The declines ledger still fails open — its failure costs one issue a human closes, which then suppresses it for good.
  • listIssues gains strict. A dedup read cannot use a list the page ceiling cut short — it answers "not filed" for everything it never reached — and a bare array can't distinguish "ended on a short page" from "ended on the ceiling". Opt-in, so no triage caller changes.
  • Reconciliation is a matching judgment, not an invention. Exact key match first (free), then one model call over the residue. Every answer is checked against the keys actually read from the control repo, so a hallucinated match files the question rather than burying it — a missed match costs one click, an accepted hallucination is a question never asked again.
  • checkSuppression's headBranchPrefix is now optional. Omitted, the PR-history read is skipped entirely rather than made and ignored: a prefix matching nothing would answer "no prior proposals" every tick for a reason no reader could tell from the feature being off.

The github fake appends every openIssue to the list issues reads back — without that, every test would pass against a run that dedups against nothing.

Config: questions-dirquestions-label / lane-label-prefix / max-new-questions. A set-and-unusable label fails the run rather than falling back; it is both the filter the read applies and the label the write applies, and a comma makes those two different things with nothing erroring.

Verification

pnpm typecheck clean, pnpm lint clean, oxfmt --check clean on all 13 changed files. 60 run tests + 24 github-app issue tests pass; full suite 2236 passed. The 5 failing files in packages/demo-agent are missing optional deps (@effect/ai, puppeteer-core, pngjs) and fail identically on a clean tree.

Follow-ups

\ Tag Action Why
1 ✍️ [SignOff] Run infra/maintenance-loop/sync-config.sh --apply in fractalboxdev/org after this deploys It pushes the three new keys and deletes org-spec-audit.questions-dir; deleting before this ships would silently move the daily file to the default maintenance/questions
2 ✍️ [SignOff] Close org#147 and org#148 unmerged after the first issue-shaped sweep, with no declined.jsonl entries They hold five unanswered questions and nothing else does until the sweep files them

…e artifact

The sweep filed a dated `<date>.md` in a draft PR every morning. Two sweeps
two days apart asked three of the same questions twice — two of them under a
byte-identical `maintenance-key` (org#147, org#148).

An identical key that re-proposes rules out the obvious cause and leaves the
real one: **the ledger could only remember "no", never "asked".** Both
suppression rules key off a TERMINATED proposal — declined in the ledger, or a
PR closed unmerged — and the first PR was neither. It was open, unanswered, in
the queue, which the design read as no signal at all. The unstable key was the
second cause and would have fixed one duplicate of the three.

So the unit is the question and the artifact is a GitHub issue. open = asked
(write nothing), closed = decided (never re-file, never reopen), absent = new.

- `createIssue` on the github-app surface, `openIssue` on `GithubService`. It
  FAILS rather than degrading to a logged no-op, unlike `openDraftPullRequest`:
  a PR write that no-ops leaves its content in git and its branch idempotent,
  while a no-op here drops the only copy of the question.
- `listIssues` gains `strict`, which throws instead of returning a list the
  page ceiling cut short. A dedup read cannot use a partial list — it answers
  "not filed" for everything it never reached — and a bare array cannot
  distinguish "ended on a short page" from "ended on the ceiling".
- `IssueRef.closedAt` — the column `pulls` lacks and `issues` has.
- Reconciliation: exact key match first, then ONE model call over the residue
  asking "one of these, or new?". Every answer is checked against the keys
  actually read from the control repo, so a hallucinated match files the
  question rather than silently burying it. The prompt now asks for noun-phrase
  keys (`pipeline-dags`, not `adopt-pipeline-dags`).
- The dedup read happens BEFORE the sweep and fails CLOSED, inverting the
  suppression primitive's posture: an unreadable ledger cost one duplicate PR,
  an unreadable issue set costs a duplicate of every question at once. The
  declines ledger still fails open — its failure costs one issue a human
  closes, and that close then suppresses it for good.
- `checkSuppression`'s `headBranchPrefix` is optional; omitted, the PR-history
  read is skipped entirely rather than made and ignored.
- The notice is a delta: filed today with links, standing open count, what the
  per-sweep cap held. `max-new-questions` bounds the writes; the old per-group
  rendering cap is gone.
- The github fake appends every `openIssue` to the list `issues` reads back, so
  "the second sweep sees the first sweep's issue" is testable at all.

Config: `questions-dir` retires for `questions-label`, `lane-label-prefix` and
`max-new-questions`. A label that is set-and-unusable fails the run instead of
falling back — it is both the filter the read applies and the label the write
applies, and a comma makes those two different things silently.

Design: fractalboxdev/org#150.

@flaredispatch-fractalboxdev flaredispatch-fractalboxdev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI code review — 💬 Comment

Risk tier: full · 0 critical · 2 warnings · 0 suggestions

Reviewers: security 0 · performance 1 · code-quality ⚠️ · documentation 1 · release-management ⚠️ · compliance ⚠️ · agents-md ⚠️

1. ⚠️ Warning — Strict issue reads fail permanently once the control repository exceeds 500 issues

📍 packages/github-app/src/issues.ts:255-263

'strict' uses 'MAX_PAGES_DEFAULT' (5) when the caller does not provide 'maxPages', then throws as soon as the fifth page is full. The spec-audit deduplication read enables 'strict' but does not configure a larger or adaptive page limit, so a control repository with more than 500 issues makes every sweep fail after five sequential API calls rather than completing. This will become a recurring availability/performance problem as the ledger grows; use an unbounded/adaptive strict pagination strategy or a targeted server-side query for the maintenance key.

2. ⚠️ Warning — Documented URL invariant is not enforced

📍 packages/core/src/services/github.ts:68-69

The 'IssueCreated.url' documentation says the URL is never empty on success, but 'createIssue' only validates the issue number and returns 'str(created.html_url)', which can be '""' when 'html_url' is missing or malformed. Either validate the URL and fail the create, or weaken the documentation to describe the possible empty value.

📋 View full logs & reviewed diff ↗

…e needs its url

Two findings from pr-review, both real.

`strict` inherited the library's 5-page default, so once the control repo held
500 label-filtered issues EVERY sweep would fail — permanently, five API calls
in. The ledger read now asks for 20 pages, and the error names the remedy. Read
the number as a canary: 2,000 questions asked and never closed is not a
pagination problem, it is the loop having outrun the team, and a red run there
beats one deciding on the first 500. It costs nothing until then — pagination
stops on the first short page.

`IssueCreated.url` was documented as never-empty and wasn't. A caller announces
the issue by linking it, so `createIssue` now fails on a missing `html_url` the
same way it fails on a missing number, rather than publishing a link to nowhere.

The fake records `maxPages` and the run test asserts the raised ceiling, so a
later edit dropping it cannot silently restore the five-page failure.
@debuggingfuture

Copy link
Copy Markdown
Member Author

Both fixed in 5754a9d.

1. Strict reads and the 5-page default — correct, and the failure is worse than described: it is permanent. Once the label-filtered set exceeds 500, every sweep fails five API calls in, forever, until someone raises the number. The ledger read now passes maxPages: 20 explicitly and the error names the remedy.

I did not make it unbounded or adaptive, and the reasoning is worth stating: this read's value is completeness, so the only alternatives are "paginate until exhausted" (unbounded work on a scheduled job) or "decide on a partial list" (the bug the flag exists to prevent). A high explicit ceiling keeps the third option — fail loudly — and costs nothing until reached, since pagination stops on the first short page. 2,000 questions asked and never closed is not a pagination problem; it is the loop having outrun the team, which the process doc names as this design's actual failure mode. A red run there is closer to correct than a silent partial read.

Search-by-key was the other candidate and I rejected it on indexing lag: repo:x label:y "maintenance-key: …" in:body is exact and cheap, but GitHub's search index is eventually consistent, and a dedup read that can miss a recently-filed issue re-files it.

2. IssueCreated.url documented as never-empty and wasn't. Correct. createIssue now fails on a missing html_url the same way it fails on a missing number — the caller announces the issue by linking it, so "" publishes a broken link rather than failing.

Also pinned both: the fake records maxPages and the run test asserts the raised ceiling, so a later edit dropping it cannot silently restore the five-page failure. New test covers the empty-url create.

@debuggingfuture
debuggingfuture merged commit 37d290f into main Aug 17, 2026
4 checks passed
@debuggingfuture
debuggingfuture deleted the org-spec-audit/questions-as-issues branch August 17, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant